
            <!DOCTYPE html>
            <html lang="en">
            <head>
                <meta charset="UTF-8">
                <title>Prometheus Operator 实战 监控 etcd 集群</title>
            </head>
            <body>
            <a href="https://andyoung.blog.csdn.net">原作者博客</a>
            <div id="content_views" class="markdown_views prism-atom-one-light">
                    <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                        <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                    </svg>
                    <p><a href="https://blog.csdn.net/agonie201218/article/details/127965449">上节课和大家讲解了 Prometheus Operator 的安装和基本使用方法</a>，这节课给大家介绍如何在 Prometheus Operator 中添加一个自定义的监控项。</p> 
<p>除了 Kubernetes 集群中的一些资源对象、节点以及组件需要监控，有的时候我们可能还需要根据实际的业务需求去添加自定义的监控项，添加一个自定义监控的步骤也是非常简单的。</p> 
<ul><li>第一步建立一个 ServiceMonitor 对象，用于 Prometheus 添加监控项</li><li>第二步为 ServiceMonitor 对象关联 metrics 数据接口的一个 Service 对象</li><li>第三步确保 Service 对象可以正确获取到 metrics 数据</li></ul> 
<p>接下来我们就来为大家演示如何添加 etcd 集群的监控。</p> 
<p>无论是 Kubernetes 集群外的还是使用 Kubeadm 安装在集群内部的 etcd 集群，我们这里都将其视作集群外的独立集群，因为对于二者的使用方法没什么特殊之处。</p> 
<h4><a id="etcd__12"></a>etcd 证书</h4> 
<p>对于 etcd 集群一般情况下，为了安全都会开启 https 证书认证的方式，所以要想让 Prometheus 访问到 etcd 集群的监控数据，就需要提供相应的证书校验。</p> 
<p>由于我们这里演示环境使用的是 Kubeadm 搭建的集群，我们可以使用 kubectl 工具去获取 etcd 启动的时候使用的证书路径：</p> 
<pre><code class="prism language-shell">$ kubectl get pods <span class="token parameter variable">-n</span> kube-system
NAME                                          READY     STATUS    RESTARTS   AGE
etcd-master                                   <span class="token number">1</span>/1       Running   <span class="token number">0</span>          2h
$ kubectl get pod etcd-master <span class="token parameter variable">-n</span> kube-system <span class="token parameter variable">-o</span> yaml
<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>
spec:
  containers:
  - command:
    - etcd
    - --peer-cert-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/peer.crt
    - --listen-client-urls<span class="token operator">=</span>https://127.0.0.1:2379
    - --advertise-client-urls<span class="token operator">=</span>https://127.0.0.1:2379
    - --client-cert-auth<span class="token operator">=</span>true
    - --peer-client-cert-auth<span class="token operator">=</span>true
    - --data-dir<span class="token operator">=</span>/var/lib/etcd
    - --cert-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/server.crt
    - --key-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/server.key
    - --trusted-ca-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/ca.crt
    - --peer-key-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd-amd64:3.1.12
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - <span class="token parameter variable">-ec</span>
        - <span class="token assign-left variable">ETCDCTL_API</span><span class="token operator">=</span><span class="token number">3</span> etcdctl <span class="token parameter variable">--endpoints</span><span class="token operator">=</span><span class="token number">127.0</span>.0.1:2379 <span class="token parameter variable">--cacert</span><span class="token operator">=</span>/etc/kubernetes/pki/etcd/ca.crt
          <span class="token parameter variable">--cert</span><span class="token operator">=</span>/etc/kubernetes/pki/etcd/healthcheck-client.crt <span class="token parameter variable">--key</span><span class="token operator">=</span>/etc/kubernetes/pki/etcd/healthcheck-client.key
          get foo
      failureThreshold: <span class="token number">8</span>
      initialDelaySeconds: <span class="token number">15</span>
      periodSeconds: <span class="token number">10</span>
      successThreshold: <span class="token number">1</span>
      timeoutSeconds: <span class="token number">15</span>
    name: etcd
    resources: <span class="token punctuation">{<!-- --></span><span class="token punctuation">}</span>
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>
  tolerations:
  - effect: NoExecute
    operator: Exists
  volumes:
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>
</code></pre> 
<p>我们可以看到 etcd 使用的证书都对应在节点的 /etc/kubernetes/pki/etcd 这个路径下面，所以首先我们将需要使用到的证书通过 secret 对象保存到集群中去：(在 etcd 运行的节点)</p> 
<pre><code class="prism language-shell">$ kubectl <span class="token parameter variable">-n</span> monitoring create secret generic etcd-certs --from-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file<span class="token operator">=</span>/etc/kubernetes/pki/etcd/ca.crt
secret <span class="token string">"etcd-certs"</span> created
</code></pre> 
<blockquote> 
 <p>如果你是独立的二进制方式启动的 etcd 集群，同样将对应的证书保存到集群中的一个 secret 对象中去即可。</p> 
</blockquote> 
<p>然后将上面创建的 etcd-certs 对象配置到 prometheus 自定义资源对象中，直接更新 prometheus 资源对象即可：</p> 
<pre><code class="prism language-shell">$ kubectl edit prometheus k8s <span class="token parameter variable">-n</span> monitoring
</code></pre> 
<p>添加挂载 secrets ：</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">nodeSelector</span><span class="token punctuation">:</span>
  <span class="token key atrule">beta.kubernetes.io/os</span><span class="token punctuation">:</span> linux
<span class="token key atrule">replicas</span><span class="token punctuation">:</span> <span class="token number">2</span>
<span class="token key atrule">secrets</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> etcd<span class="token punctuation">-</span>certs
</code></pre> 
<p>更新完成后，我们就可以在 Prometheus 的 Pod 中获取到上面创建的 etcd 证书文件了，具体的路径我们可以进入 Pod 中查看：</p> 
<pre><code class="prism language-shell">$ kubectl <span class="token builtin class-name">exec</span> <span class="token parameter variable">-it</span> prometheus-k8s-0 /bin/sh <span class="token parameter variable">-n</span> monitoring
Defaulting container name to prometheus.
Use <span class="token string">'kubectl describe pod/prometheus-k8s-0 -n monitoring'</span> to see all of the containers <span class="token keyword">in</span> this pod.
/ $ <span class="token function">ls</span> /etc/prometheus/secrets/etcd-certs/
ca.crt      healthcheck-client.crt  healthcheck-client.key
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/54775d488a2e17c97d114f63b4a406ef.png" alt="image-20221122132315434"></p> 
<h4><a id="_ServiceMonitor_116"></a>创建 ServiceMonitor</h4> 
<p>现在 Prometheus 访问 etcd 集群的证书已经准备好了，接下来创建 ServiceMonitor 对象即可（prometheus-serviceMonitorEtcd.yaml）</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> monitoring.coreos.com/v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> ServiceMonitor
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
  <span class="token key atrule">name</span><span class="token punctuation">:</span> etcd<span class="token punctuation">-</span>k8s
  <span class="token key atrule">namespace</span><span class="token punctuation">:</span> monitoring
  <span class="token key atrule">labels</span><span class="token punctuation">:</span>
    <span class="token key atrule">k8s-app</span><span class="token punctuation">:</span> etcd<span class="token punctuation">-</span>k8s
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
  <span class="token key atrule">jobLabel</span><span class="token punctuation">:</span> k8s<span class="token punctuation">-</span>app
  <span class="token key atrule">endpoints</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> <span class="token key atrule">port</span><span class="token punctuation">:</span> port
    <span class="token key atrule">interval</span><span class="token punctuation">:</span> 30s
    <span class="token key atrule">scheme</span><span class="token punctuation">:</span> https
    <span class="token key atrule">tlsConfig</span><span class="token punctuation">:</span>
      <span class="token key atrule">caFile</span><span class="token punctuation">:</span> /etc/prometheus/secrets/etcd<span class="token punctuation">-</span>certs/ca.crt
      <span class="token key atrule">certFile</span><span class="token punctuation">:</span> /etc/prometheus/secrets/etcd<span class="token punctuation">-</span>certs/healthcheck<span class="token punctuation">-</span>client.crt
      <span class="token key atrule">keyFile</span><span class="token punctuation">:</span> /etc/prometheus/secrets/etcd<span class="token punctuation">-</span>certs/healthcheck<span class="token punctuation">-</span>client.key
      <span class="token key atrule">insecureSkipVerify</span><span class="token punctuation">:</span> <span class="token boolean important">true</span>
  <span class="token key atrule">selector</span><span class="token punctuation">:</span>
    <span class="token key atrule">matchLabels</span><span class="token punctuation">:</span>
      <span class="token key atrule">k8s-app</span><span class="token punctuation">:</span> etcd
  <span class="token key atrule">namespaceSelector</span><span class="token punctuation">:</span>
    <span class="token key atrule">matchNames</span><span class="token punctuation">:</span>
    <span class="token punctuation">-</span> kube<span class="token punctuation">-</span>system
</code></pre> 
<p>上面我们在 monitoring 命名空间下面创建了名为 etcd-k8s 的 ServiceMonitor 对象，基本属性和前面章节中的一致，匹配 kube-system 这个命名空间下面的具有 k8s-app=etcd 这个 label 标签的 Service，jobLabel 表示用于检索 job 任务名称的标签，和前面不太一样的地方是 endpoints 属性的写法，配置上访问 etcd 的相关证书，endpoints 属性下面可以配置很多抓取的参数，比如 relabel、proxyUrl，tlsConfig 表示用于配置抓取监控数据端点的 tls 认证，由于证书 serverName 和 etcd 中签发的可能不匹配，所以加上了 insecureSkipVerify=true</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/ef765765947f0a61b9d873ac97fa9108.jpeg" alt="tlsConfig">tlsConfig</p> 
<blockquote> 
 <p>关于 ServiceMonitor 属性的更多用法可以查看文档：https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md 了解更多</p> 
</blockquote> 
<p>直接创建这个 ServiceMonitor 对象：</p> 
<pre><code class="prism language-shell">$ kubectl create <span class="token parameter variable">-f</span> prometheus-serviceMonitorEtcd.yaml
servicemonitor.monitoring.coreos.com <span class="token string">"etcd-k8s"</span> created
</code></pre> 
<h4><a id="_Service_160"></a>创建 Service</h4> 
<p>ServiceMonitor 创建完成了，但是现在还没有关联的对应的 Service 对象，所以需要我们去手动创建一个 Service 对象（prometheus-etcdService.yaml）：</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Service
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
  <span class="token key atrule">name</span><span class="token punctuation">:</span> etcd<span class="token punctuation">-</span>k8s
  <span class="token key atrule">namespace</span><span class="token punctuation">:</span> kube<span class="token punctuation">-</span>system
  <span class="token key atrule">labels</span><span class="token punctuation">:</span>
    <span class="token key atrule">k8s-app</span><span class="token punctuation">:</span> etcd
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
  <span class="token key atrule">type</span><span class="token punctuation">:</span> ClusterIP
  <span class="token key atrule">clusterIP</span><span class="token punctuation">:</span> None
  <span class="token key atrule">ports</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> port
    <span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">2379</span>
    <span class="token key atrule">protocol</span><span class="token punctuation">:</span> TCP

<span class="token punctuation">---</span>
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Endpoints
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
  <span class="token key atrule">name</span><span class="token punctuation">:</span> etcd<span class="token punctuation">-</span>k8s
  <span class="token key atrule">namespace</span><span class="token punctuation">:</span> kube<span class="token punctuation">-</span>system
  <span class="token key atrule">labels</span><span class="token punctuation">:</span>
    <span class="token key atrule">k8s-app</span><span class="token punctuation">:</span> etcd
<span class="token key atrule">subsets</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">addresses</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> <span class="token key atrule">ip</span><span class="token punctuation">:</span> 10.151.30.57
    <span class="token key atrule">nodeName</span><span class="token punctuation">:</span> etc<span class="token punctuation">-</span>master
  <span class="token key atrule">ports</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> port
    <span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">2379</span>
    <span class="token key atrule">protocol</span><span class="token punctuation">:</span> TCP
</code></pre> 
<p>我们这里创建的 Service 没有采用前面通过 label 标签的形式去匹配 Pod 的做法，因为前面我们说过很多时候我们创建的 etcd 集群是独立于集群之外的，这种情况下面我们就需要自定义一个 Endpoints，要注意 metadata 区域的内容要和 Service 保持一致，Service 的 clusterIP 设置为 None，对改知识点不太熟悉的，可以去查看我们前面关于 Service 部分的讲解。</p> 
<p>Endpoints 的 subsets 中填写 etcd 集群的地址即可，我们这里是单节点的，填写一个即可，直接创建该 Service 资源：</p> 
<pre><code class="prism language-shell">$ kubectl create <span class="token parameter variable">-f</span> prometheus-etcdService.yaml
</code></pre> 
<p>创建完成后，隔一会儿去 Prometheus 的 Dashboard 中查看 targets，便会有 etcd 的监控项了：</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/8643e00530437f572cffd7c4e13898cd.jpeg" alt="prometheus etcd">prometheus etcd</p> 
<p>可以看到还是有一个明显的错误，和我们上节课监控 kube-scheduler 的错误比较类似于，因为我们这里的 etcd 的是监听在 127.0.0.1 这个 IP 上面的，所以访问会拒绝：</p> 
<pre><code class="prism language-shell">--listen-client-urls<span class="token operator">=</span>https://127.0.0.1:2379
</code></pre> 
<p>同样我们只需要在 /etc/kubernetes/manifest/ 目录下面（static pod 默认的目录）的 etcd.yaml 文件中将上面的<code>listen-client-urls</code>更改成 0.0.0.0 即可：</p> 
<pre><code class="prism language-shell">--listen-client-urls<span class="token operator">=</span>https://0.0.0.0:2379
</code></pre> 
<p>重启 etcd，生效后，查看 etcd 这个监控任务就正常了：</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/0e76e4fe2e142e211c47d908d3b11f2b.jpeg" alt="prometheus etcd">prometheus etcd</p> 
<p>数据采集到后，可以在 grafana 中导入编号为<code>3070</code>的 dashboard，获取到 etcd 的监控图表。</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/08fb6446672d3f79e761e5e2cdb2620a.jpeg" alt="grafana etcd dashboard">grafana etcd dashboard</p>
                </div>
            </body>
            </html>
            